Toward a Theoretical Understanding of Why and WhenDecision Tree Pruning Algorithms

Recent empirical studies revealed two surprising pathologies of several common decision tree pruning algorithms. First, tree size is often a linear function of training set size, even when additional tree structure yields no increase in accuracy. Second, building trees with data in which the class label and the attributes are independent often results in large trees. In both cases, the pruning algorithms fail to control tree growth as one would expect them to. We explore this behavior theoretically by constructing a statistical model of reduced error pruning. The model explains why and when the pathologies occur, and makes predictions about how to lessen their eeects. The predictions are operationalized in a variant of reduced error pruning that is shown to control tree growth far better than the original algorithm.